Machine learning PhD students are in a unique position: they often need to run large-scale experiments to conduct state-of-the-art research but they don't have the support of the platform teams that industrial ML engineers can rely on. As former PhD students ourselves, we recount our hands-on experience with these challenges and explain how open-source tools like Determined would have made grad school a lot less painful. When we started graduate school as PhD students at Carnegie Mellon University (CMU), we thought the challenge laid in having novel ideas, testing hypotheses, and presenting research. Instead, the most difficult part was building out the tooling and infrastructure needed to run deep learning experiments. While industry labs like Google Brain and FAIR have teams of engineers to provide this kind of support, independent researchers and graduate students are left to manage on their own.

Jacob Steinhardt (Google Scholar) (Website) is an assistant professor at UC Berkeley. His main research interest is in designing machine learning systems that are reliable and aligned with human values. Some of his specific research directions include robustness, rewards specification and reward hacking, as well as scalable alignment. His most recent paper at ICLR 2021 proposes a new test to measure an NLP model's accuracy on a wide variety of tasks, ranging from mathematics, US history, law, and more. It provides a measurement tool to help researchers specify an important problem: while current models can achieve superhuman performance on benchmarks, they lack the ability to understand language on a whole. Another of Jacob's papers at ICLR focuses on measuring a language model's knowledge of basic concepts of morality. It shows that current language models have a promising but incomplete ability to predict basic human ethical judgements. "Test accuracy is a very limited metric." "You might not be able to get lots of feedback on human values." Below are the show notes and full transcript. As always, please feel free to reach out with feedback, ideas, and questions! I think it required me to learn to become a significantly better writer. And I think that helped later on, because it made me feel more comfortable pursuing unusual ideas. I knew I had the skills to present those ideas. As long as I believed in them, I could get other people to believe in them." You just want this very diverse distribution of things that are deeply ingrained in evolutionary history as opposed to being part of explicit reasoning" First of all, test accuracy is a very limited metric. What are we trying to do with it? For a while, there was a lot of climate skepticism or climate denial. At some point it becomes pretty clear, when there's regular heat waves fires and that sort of thing. You probably wanted to do something about it before that point. Having these more subtle measurements that you can look at are important. And the other thing is I think it actually laid the groundwork for the more extreme weather events to become a convincing signal. Jacob Steinhardt: Another thing that I'm interested in is just measuring the progress in capabilities, getting different AI capabilities seems important. Vision tasks just seem to be falling like flies. I don't know if there's any vision tasks that's survived for more than a year and a few tasks seem a little bit better, but I think those are also starting to fall like flies. I know we've come up with a few harder tasks. ML Systems are still not very good at math. Humans also aren't very good at math, but also not good at law it turns out.

Purvanshi Mehta shares her invaluable insights into the importance of finding your niche in Machine Learning and how best to go about it.

Originally, Maysam Moussalem dreamed of being an architect. "When I was 10, I looked up to see the Art Nouveau dome over the Galeries Lafayette in Paris, and I knew I wanted to make things like that," she says. "Growing up between Austin, Paris, Beirut and Istanbul just fed my love of architecture." But she found herself often talking to her father, a computer science (CS) professor, about what she wanted in a career. "I always loved art and science and I wanted to explore the intersections between fields. CS felt broader to me, and so I ended up there."

I walked into my adviser's office, overflowing with frustration and confusion about the advice I had received at a recent career development workshop. It reiterated what I had heard so many times before: I should follow my dream, and if I didn't yet know what that was, I should live with career uncertainty until I figured it out. But as an international student working in the United States, taking time to explore wasn't an option for me. After listening to me rant, my adviser calmly looked across his desk. He told me that instead of focusing on finding a dream job, I should think about what I am good at and what makes me happy at least 80% of the time. This advice surprised me at first, but it ended up being exactly what I needed to hear. > “I should think about what I am good at and what makes me happy at least 80% of the time.” I had spent the previous 22 years following my childhood dream—becoming a professor of marine biology. However, in grad school I saw how applying for grants is a constant source of worry for many professors. I realized I did not want to be responsible for the salaries of my hypothetical lab members. About 4 years into the program, I decided I did not want to pursue a career in research after all. I began to attend career panels, which all followed a worryingly similar template. I would walk into the room with other excited graduate students and collect my free cookies and coffee, confident that the panelists would have the magical answers I needed. Instead, they would talk—again—about following their dreams. The message: I just needed to find a new dream. It would mean taking time off from work to self-reflect and discover a new path. But I couldn't stay in the country without a visa. For most academic researchers, obtaining a university-sponsored visa is relatively straightforward. But outside of academia, it is infinitely more complex, requiring a company that has a job opening and is willing to foot the bill for a work visa. As well-meaning as the panelists were, they fell silent when I brought up this dilemma. I felt totally lost. Finally, I went to my adviser for help. We hadn't talked much about my career plans over the years, but I felt I needed a new perspective from someone who knew me well. When he offered his advice, I was taken aback at first. What happened to “if you love what you do, you'll never work a day in your life”? My adviser assured me there is seldom such a job. Every job has its ugly bits. But as long as you're happy most of the time, you can struggle through the parts you don't like. He also said it was important to find a job I was good at, especially because my visa applications required me to make the case that I would benefit the country. I was relieved to finally have helpful, practical advice. But I discovered that finding overlap between what I like and what I'm good at was not easy. I love scuba diving, but the physical demands are a challenge for me. I'm good at teaching, as evidenced by my friends nagging me to teach them chemistry and microbiology during my high school and undergraduate years and getting rave reviews from my students when I was a teaching assistant, but I don't like repeating the same content every year. Through my teaching experience, however, I also learned that I love telling stories about science. Maybe science communication would offer the overlap I was looking for. To test the waters, during my “spare time” in grad school I started a blog about the history of scientific discoveries. I found that I loved the freedom to choose what to write about, and I never encountered a challenge I didn't enjoy. As for whether I was any good at it, the signs were promising. My writing got noticed, eventually by people at my institution, and I was given opportunities to write press releases and stories for the university's news bureau. After 3 years of writing, I was offered a position as a science writer. It's nothing like my childhood dream. But I am happy—more than 80% of the time.

Machine learning PhD students are in a unique position: they often need to run large-scale experiments to conduct state-of-the-art research but they don't have the support of the platform teams that industrial ML engineers can rely on. As former PhD students ourselves, we recount our hands-on experience with these challenges and explain how open-source tools like Determined would have made grad school a lot less painful. When we started graduate school as PhD students at Carnegie Mellon University (CMU), we thought the challenge laid in having novel ideas, testing hypotheses, and presenting research. Instead, the most difficult part was building out the tooling and infrastructure needed to run deep learning experiments. While industry labs like Google Brain and FAIR have teams of engineers to provide this kind of support, independent researchers and graduate students are left to manage on their own.

Harvey Mudd computer science professor Jim Boerkoel works with a student in his robotics lab, where ... [ ] he focuses on using AI to develop human-robot teamwork. The rapid expansion of artificial intelligence from facial recognition and self-driving cars to understanding human speech is having a major impact on business and society, which is why the lack of diversity among the people developing AI tools is so troubling. A recent study published by the AI Now Institute of New York University concluded that a "diversity disaster" has resulted in flawed AI systems that perpetuate gender and racial biases. The report found that more than 80 percent of AI professors are men and only 15% of AI researchers at Facebook and 10 percent of AI researchers at Google are women. The numbers reflect a larger issue facing the computer sciences where, in 2018, less than 25 percent of PhDs were awarded to females and/or minorities, who are historically underrepresented in computing. Industry and academia are taking steps to increase diversity among AI researchers through steps designed to ensure that future technology benefits all people and not just a homogenous group of white males.

EPISODE SUMMARY Sherrie Wang, a fourth year PhD student at Stanford's Institute for Computational and Mathematical Engineering (ICME), explains how she applies machine learning methods to help solve global food security challenges. EPISODE NOTES Sherrie brings an interdisciplinary and entrepreneurial perspective to her research that she has developed through her work in the fields of computational finance, biomedical engineering, and computer vision. Sherrie explains that about 1 in 9 people do not have access to adequate food. She is using satellite imagery and machine learning to identify and map crops around the world, see where people are most vulnerable, and what interventions or policies have the greatest effect. "There are a lot of problems that technology alone can't solve and we still need to understand the roots of a lot of the problems. That involves talking to people in the earth sciences and agriculture and learning from them. In those conversations, data science just becomes a tool, a very useful tool but it's a tool in the context of some much larger problem," she explained to Stanford's Margot Gerritsen, Stanford professor and host of the Women in Data Science podcast.

They're vast or complex or both, and we can't analyze them without help. Specifically, help from self-improving machine learning algorithms. These algorithms can glean "insights into how the world works that a person wouldn't be able to see, because they're [too] abstract or [too] fine-grained," says Meghan Hickey, a Boston-based machine learning engineer at Pryon. That can mean picking up on patterns humans can't see -- like learning to spot cancer symptoms invisible to the human eye -- or performing human analysis at nonhuman speeds. The company makes a data analytics platform called Unify that clients often use to integrate overlapping datasets. For example, if two merging banks share some clients, they want to make sure those clients aren't double-entered in their respective databases.

I'm an enthusiastic realist when it comes to technology. What motivates me is not the machine, but what it can do to help us to solve problems. More sophisticated machines allow us to tackle more complex problems. You may have heard that AI is going to solve all our problems. Or did you hear that it's going to steal our jobs, enslave and then destroy us?